Blog post about agentic pipeline using valkey #290

vitarb · 2025-07-01T23:20:54Z

Description

This PR adds a full end-to-end agentic demo built on Valkey, showcasing a real-time, stream-driven pipeline for ingesting, enriching, and distributing news content to personalized user feeds. It includes:

A Twitter-style agentic architecture implemented with Valkey Streams, Lua scripting, and Prometheus metrics.

GPU‑accelerated enrichment pipeline using HuggingFace Transformers.

Real-time observability via Grafana dashboards.

Fully working Docker Compose setup with support for both CPU and GPU execution.

A blog post documenting the system design, performance characteristics, and real-world debugging lessons.

Issues Resolved

Closes valkey-io/valkey-doc#320

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.

stockholmux

I can't review this as-is - it lacks the requisite frontmatter meta data, so I can tell it hasn't been previewed. Please make sure your post works in zola before submitting a PR (see the README)

stockholmux · 2025-07-23T21:19:05Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

@@ -0,0 +1,177 @@
+# Lightning-Fast Agent Messaging with Valkey


Missing front matter.

Also - you need a bio.

I know, left that for the end :)

stockholmux · 2025-07-23T21:25:20Z

Also: please fix the DCO issue.

makubo-aws · 2025-07-23T21:46:17Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+**4. Readers fell behind during spikes**
+Fixed 50 pops/sec couldn’t keep up with 10k users. Self-tuning delay (`latest_uid * POP_RATE`) scaled up to 200 pops/sec.
+
+All fixes are now defaults in the repo.


repository. Let's avoid shorthand. Also, this is the first mention of the repo. I would introduce it earlier in the blog.

First mention now reads valkey‑agentic‑demo repository with hyperlink in intro.

makubo-aws · 2025-07-23T21:47:17Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+
+Modern applications are moving beyond monoliths into distributed fleets of specialized agents—small programs that sense, decide, and act in real-time. When hundreds of these interact, their messaging layer must be lightning-fast, observable, and flexible enough to evolve without rewrites.
+
+That requirement led us to **Valkey**: an open-source, community-driven, in-memory database fully compatible with Redis. With streams, Lua scripting, a mature JSON & Search stack, and a lightweight extension system, Valkey provides our agents with a fast, shared nervous system.


Who is "us"? The wording here sounds like a 3rd party considering different datastore options and discovered Valkey. But then later in the blog you say "our roadmap", as a valkey member, which I think is what you mean to refer to.

Adopted consistent perspective: we = engineering team using Valkey, not Valkey core maintainers.

makubo-aws · 2025-07-23T21:52:00Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+
+Tails the per-user stream and emits new entries directly to the browser.
+
+### Self-Tuning Readers (load generator & consumer)


Added Stage 5 – Reader (self‑tuning load consumer) and included code.

makubo-aws · 2025-07-23T22:21:15Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

@@ -0,0 +1,177 @@
+# Lightning-Fast Agent Messaging with Valkey
+
+## From Tweet to Tailored Feed


I love this intro so far, because it's highlighting Valkey's strengths and the way it addresses modern application requirements. I would add a bit more context about what you will cover in the blog and what the reader can expect to get out of it. Using your content and the help of AI, I got the following example of what I mean, which I liked:

As modern software moves toward ecosystems of intelligent agents—small, purpose-built programs that sense, decide, and act—the infrastructure underneath needs to keep up. These systems aren’t batch-driven or monolithic. They’re always on, highly parallel, and constantly evolving. What they need is a messaging backbone that’s fast, flexible, and observable by design.

Valkey fits that role perfectly. As a Redis-compatible, in-memory data store, it brings together real-time speeds with a growing suite of tools well-suited for agentic systems. Valkey Streams offer a clean, append-only log for sequencing events. Lua scripting lets you run coordination logic server-side, where it’s faster and easier to manage. JSON and Search support structured data flows without external ETL.

In this post, we’ll put all of that into practice. We’ve built a real-time content pipeline where each news headline is ingested, enriched with topics, routed to interested users, and pushed live to their feeds—all through a series of autonomous agents communicating via Valkey. The result is a low-latency, high-throughput system that’s simple to understand and easy to scale.

You’ll see how the pieces fit together: from the first headline pulled off the wire to the last WebSocket update in the browser. Along the way, we’ll share the design trade-offs, the tuning decisions, and the fixes that helped us hit production-grade speed and reliability. All of it runs in Docker, with GPU offload if you have one, and Grafana dashboards that light up as soon as messages start flowing.

Added three‑paragraph preview summarising demo scope and learning outcomes.

makubo-aws · 2025-07-23T22:24:30Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+
+---
+
+## Why We Bet on Valkey


Similar issue as with intro. Sounds like Valkey betting on itself.

Section retitled Why We Picked Valkey for the Job and rewritten in third‑party voice.

makubo-aws · 2025-07-23T22:26:23Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+
+## Why We Bet on Valkey
+
+Valkey Streams and consumer groups move messages in <1 ms. Lua keeps fan-out logic server-side. JSON/Search allows enrichment to stay in-memory. Grafana charts latency and backlog immediately. Python agents can be swapped for Rust or Go with no changes to the datastore.


I think this section is an opportunity to expand on these value propositions of Valkey. Why is Lua scripts running server side useful? What makes JSON in Valkey great? etc.

Table now explains each feature’s concrete value (ordering, zero round‑trip Lua, in‑memory JSON/Search, INFO metrics).

makubo-aws · 2025-07-23T22:30:46Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+
+## Observability That Comes Standard
+
+Every agent exports metrics. Grafana's dashboard auto-populates:


Expand. Why did you build in observability? What is the value of each metrics? Is there something specific about Valkey's design that enables this?

Observability table now ties every metric to an operational question.

makubo-aws · 2025-07-23T22:31:18Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+| p99 Valkey op latency      | ≈ 200 µs               |
+| GPU uplift (A10G)          | 5x faster enrichment |
+
+Scaling up? One Docker command. No Helm. No YAML deep dives.


Please expand. No short hand

makubo-aws · 2025-07-23T22:32:48Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

+* **Rust agents** using the same Streams API but with lower memory.
+* **Auto-provisioned ACLs & metrics** via the MCP server.
+
+Pull requests and fresh ideas welcome.


Feed UI & Grafana URLs are explicit Markdown links; repository also hyper‑linked.

makubo-aws · 2025-07-23T22:33:41Z

content/blog/2025-07-01-valkey-powered-agentic-pipeline.md

Super cool overall, thanks for writing this up! Left some comments and suggestions

Thanks, addressed comments, please let me know if you have any other recommendations.

Blog post about agentic pipeline using valkey

81600de

vitarb requested review from madolson and stockholmux as code owners July 1, 2025 23:20

vitarb added 3 commits July 1, 2025 16:32

Add a section about EC2 execution

7b66bff

Improve blog post

9417961

Update blogpost making it more narrative driven

7814aab

hpatro assigned vitarb Jul 18, 2025

Make some edits improving readibility and flow of the article

ea098b4

stockholmux requested changes Jul 23, 2025

View reviewed changes

makubo-aws suggested changes Jul 23, 2025

View reviewed changes

Addressing review comments

983b6d3

		@@ -0,0 +1,177 @@
		# Lightning-Fast Agent Messaging with Valkey


		Modern applications are moving beyond monoliths into distributed fleets of specialized agents—small programs that sense, decide, and act in real-time. When hundreds of these interact, their messaging layer must be lightning-fast, observable, and flexible enough to evolve without rewrites.

		That requirement led us to Valkey: an open-source, community-driven, in-memory database fully compatible with Redis. With streams, Lua scripting, a mature JSON & Search stack, and a lightweight extension system, Valkey provides our agents with a fast, shared nervous system.


		Tails the per-user stream and emits new entries directly to the browser.

		### Self-Tuning Readers (load generator & consumer)

		@@ -0,0 +1,177 @@
		# Lightning-Fast Agent Messaging with Valkey

		## From Tweet to Tailored Feed


		## Why We Bet on Valkey

		Valkey Streams and consumer groups move messages in <1 ms. Lua keeps fan-out logic server-side. JSON/Search allows enrichment to stay in-memory. Grafana charts latency and backlog immediately. Python agents can be swapped for Rust or Go with no changes to the datastore.


		## Observability That Comes Standard

		Every agent exports metrics. Grafana's dashboard auto-populates:

Blog post about agentic pipeline using valkey #290

Are you sure you want to change the base?

Blog post about agentic pipeline using valkey #290

Uh oh!

Conversation

vitarb commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

stockholmux left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stockholmux commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vitarb commented Jul 1, 2025 •

edited

Loading

stockholmux commented Jul 23, 2025 •

edited

Loading